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IMAGE PROCESSING DEVICE, PROGRAM PRODUCT AND METHOD 

[0001] This application is based on Japanese Patent 

Application No. 2000-267304 filed on September A, 2000, the 
contents of which are hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 
1. Field of the Invention 

[0002] The present invention relates to an image processing 
device, its program product and method. In particular, the 
invention relates particularly to an image processing device, 
its program product and method for synthesizing foreground 
p image data consisting of character or graphic images with 

y background image data. 

u 

□ 2 . Description of the Related Art 

[0003] The presentation materials that are printed on 
overhead projector sheets often contain foreground image data 
containing character images, etc., placed in front of 
background image data. 

[0004] However, there are cases where the boundaries of 
the foreground image data and the background image data become 
unidentifiable depending on the colors of the background image 
data that exist in the areas surrounding the foreground image 
data such as character images, etc. and consequently, the 
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character images, etc, stop being conspicuous completely. 
[0005] As a means to solve such a problem, a technology 
has been known in which the colors of the foreground image 
data such as character images are changed depending on the 
colors of the background image data in order to make the 
foreground image data such as character images stand out among 
the background image data. 

[0006] For example. Publication of Unexamined Japanese 
Patent Application, JP-A-10-293835 discloses a technology 
of adjusting colors of character images as character images 
appear differently depending on background image data when 
character images and their background image data are 
synthesized. Publication of Unexamined Japanese Patent 
hj Application, JP-A-7-256972 discloses a technology , for a case 

P when character images are synthesized with another images 

H 

and the density difference between the both images is within 
a certain preset value, of changing the density of at least 
one of the both images in the areas where the both images 
overlap each other. 

[0007] However, in those technologies mentioned above, 
since the color adjustment is done independently to each area 
or pixel of character images, there can be a case where 
character images , which appear as the same color on the original 
document, are adjusted to different colors depending on the 
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colors of the background image data. 

[ 0008 ] Therefore , it has been a problem that , when character 
images obtained by reading an original document on which 
characters are color-coded depending on the function of each 
character, such as a part of title, subtitle, and keyword, 
are synthesized with the background image data, 
color-adjusted character images after synthesization lose 
the color-coding information completely. 

SUMMARY OF THE INVENTION 
[0009] It is an object of the present invention to provide 



U an improved image processing device, program product and 



method for solving various problems mentioned above. 



y [0010] It is a further object of the present invention to 

P provide an image proces s ing device , program product and method 

for securely identifying character images, etc., embedded 
in background image data without damaging color-coding 
information such as a title expressed in a uniform color on 
the foreground image data, in synthesizing foreground image 
data such as character images and background image data. 
[0011] According to an aspect of the invention, it is an 
image processing device comprising: a first color detection 
means for detecting colors of a first dLmage data by each 
processing unit; a second color detection means for detecting 
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colors of a second image data that serves as the first image 
data's background by each processing unit; and a color 
adjusting means for specifying a uniform adjusting color that 
makes the first image data recognizable against all colors 
of the second image data that serves as the first image data's 
background, concerning the first image data that have 
approximately equal colors. 

[0012] According to another aspect of the invention, it 
Q is a program product for imageprocessing that causes a computer 

to execute a process comprising the steps of : detecting colors 

m 

of a first image data by each processing unit ; detecting colors 
yj of a second image data that serves as the first image data's 

p background by each processing unit; and specifying a uniform 

i 

yl adjusting color that makes the first image data recognizable 

2 

Cj against all colors of the second image data that serves as 

the first image data's background, concerning the first image 
data that have approximately equal colors . 
[0013] According to still another aspect of the invention, 
it is an image processing method comprising the steps of: 
detecting colors of a first image data by each processing 
unit; detecting colors of a second image data that serves 
as the first image data's background by each processing unit; 
and specifying a uniform adjusting color that makes the first 
image data recognizable against all colors of the second image 
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data that serves as the first image data's background, 
concerning the first image data that have approximately equal 
colors . 

[0014] The objects, features, and characteristics of this 
invention other than those set forth above will become apparent 
from the description given herein below with reference to 
preferred embodiments illustrated in the accompanying 
drawings . 

P 

^ BRIEF DESCRIPTION OF THE DRAWINGS 

Ai ... 

[0015] Fxg. 1 IS a block diagram showing the general 

SI 

constitution of an image processing device according to an 
Q embodiment of the invention; 

y [0016] Fig. 2 is a flow chart showing the image processing 

P 

□ of an image processing unit; 

[0017] Fig. 3 is a flow chart showing the procedure of a 
synthesizing process; 

[0018] Fig. 4 is an example of background image data; 
[0019] Fig. 5 is an example of foreground image data; 
[0020] Fig. 6 is a diagram describing how background colors 
are detected; 

[0021] Fig. 7 is a diagram showing a correlation table; 
and 

[0022] Fig. 8 is an example of synthesizing image data. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
[0023] The embodiments of this invention will be described 
below with reference to the accompanying drawings . 
[0024] Fig. 1 is a block diagram showing the general 
constitution of an image processing device according to an 
embodiment of the invention. 

[0025] The image processing device has an image processing 
unit 10/ an operating unit 20, a controller 30, an image input 
unit 40, and an image output unit 50. 

[0026] The image processing unit 10 has a character 



bJ recognition unit 11, an area separation unit 12, a bitmap 

O processing unit 13 , a vector conversion unit 14 , a binarization 

W unit 15, a synthesizing unit 16, a format conversion unit 

^ 17, and a memory 18, and is capable of applying a specified 

process to inputted foreground image data and synthesizing 
the foreground image data and the background image data. 
[ 0027 ] Although it is not shown in the drawing , the operating 
unit 20 has a touch screen display and input keys and is used 
by the user for entering instruction information. The 
instruction information includes specifically the scanning 
start instruction for starting scanning of the document by 
the image input unit 4 0 and instructions for manual setup 
items . 
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[0028] The manual setup items include a setup for readout 
mode^ a setup for post-processing, a setup for image 
synthesizing mode, and a setup for output format. 
[0029] The readout mode includes the color mode for 

processing a document image as a color image and the 
monochromatic mode for processing a document image as a 
monochromatic image at the image input unit 40. 
[0030] The setup for post-processing is a mode for selecting 
Q various post-processes to be applied to the areas that are 

^ separated by the area separation unit 12 and the character 

m 

M recognition unit 11 of the image processing unit 10, i.e., 

yj character areas , graphics areas , and photographic areas . The 

□ post-processes include character coding at the character 

y recognition unit 11, bitmap processing at the bit map 

^ processing unit 13, vector conversion process at the vector 

conversion unit 14, and binarization process at the 

binarization unit 15. 

[0031] The image synthesizing mode includes an "image 
synthesizing-no mode," in which the inputted image data are 
not synthesized at the image processing unit 10, an "image 
synthesizing-yes mode," in which the inputted image data are 
synthesized, in other words, recognized as foreground image 
data, and a "background input mode," in which the inputted 
image data are recognized as background image data. 
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[0032] The output format is a format for output files that 
are prepared at the format conversion unit 17 of the image 
processing unit 10. The setup of this output format is 
selected from the general purpose file formats such as a 
document file format , a page description language format, 
a file format for document display, and a file format for 
storing images . 

[0033] The controller 30 has an interface 32 for the control 
unit 20, an interface 33 for the image input unit 40, an 
^ interface 34 for the image output unit 50, and a CPU 31 for 

controlling the entire image processing device, 
[0034] This controller 30 receives instruction information 
from the operating unit 20 through the interface 32. The 
information of the manual setup items thus received are 



O inputted into the image processing unit 10. When the 

controller 3 0 receives the scanning start instruction from 
the operating unit 2 0, it transmits the starting instruction 
to the image input unit 40 for the scanning operation either 
in the color mode or in the monochromatic mode depending on 
the readout mode. 

[0035] The image input unit 40 consists of an image readout 
device such as a scanner. When it receives the start 
instruction for the scanning operation from the controller 
30, the image input unit 40 reads (scans) the document in 
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the specified readout mode (color /monochromatic mode) • The 
image data obtained by reading the document is transmitted 
to the image processing unit 10 via the interface 33. 
[0036] The image output unit 50 has a first output unit 
51 and a second output unit 52, and receives the image data 
transmitted from the image processing unit 10 via the interface 
34. The first output unit 51 consists of an image forming 
device such as a color printer, and prints the image data 
2 on a recording medium such as a sheet of paper. The second 

^ output unit 52 consists of, for example, a computer equipped 

^ with a monitor, and the transmitted image data are displayed, 

M 

\jj stored or data processed. 

S [0037] Next, the operation of the image processing unit 

yj 10 will be described using the flow charts shown in Fig. 2 

O and Fig. 3. 

[0038] First, at the step Sll, character recognition 

processing will be performed for the image data inputted into 
the image processing unit 10. 

[0039] If the background input mode is not set up here, 
the character recognition unit 11 extracts character areas 
where the character images exist from the inputted image data . 
The image data , from which the character areas have been removed , 
are inputted into the area separation unit 12. 
[0040] The character recognition unit 11 recognizes 
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character codes from character images and converts them into 
character code data. The character recognition unit 11 
detects character information such as position information, 
character counts and color information from character images . 
The position information includes the x-y coordinates, width 
information, and length (height) dimensions of the character 
images. The character recognition unit 11 outputs the 
character code data to the synthesizing unit 16 together with 
thecharacter information containing the position information 
and the color information. If the binarization process is 
set up as the post-process by the user, the character area 
will be inputted into the binarization unit 15. 
Q [0041] If the background input mode is set up here, the 

y character recognition unit 11 transmits all the areas of the 

g 

inputted image data to the area separation unit 12 identifying 
them as non-character areas . 

[0042] At the step S12, the area separation process will 
be conducted to the image data, from which character images 
have been removed. 

[0043] If the background input mode is not set up here, 
the area separation unit 12 separates the image data, from 
which character images have been removed, into graphic areas 
and photographic areas. Photographic images in the 
photographic areas will be added with position information 
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and transmitted to the bitmap processing unit 13 , while graphic 
images in the graphic image areas will be added with position 
information and will be transmitted to the vector conversion 
unit 14. 

[0044] On the other hand, if the background input mode is 
set up here, the area separation unit 12 identifies all the 
areas of the inputted image data as photographic areas and 
transmits photographic images in the photographic areas to 
the bitmap processing unit 13. 

[0045] At the stepS13 , the bitmap processing unit 13 applies 
edge compensation and smoothing to the inputted photographic 
images in the photographic areas. After that, the data in 
the photographic area are inputted into the synthesizing unit 
16 with the position information. 

[0046] At the step S14, the vector conversion unit 14 
vector-converts the inputted graphic images in the graphic 
areas to generate vector data. The vector conversion unit 
14 detects attributive information including color 
information such as the color of the enclosed areas surrounded 
by the vector data, line widths, line types, line colors and 
endpoint shapes. The vector data are inputted into the 
synthesizing unit 16 together with the attributive 
information. The vector conversion means converting 
graphics consisting of mesh points into vector data such as 
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straight lines, arcs, and Bezier curves. 

[0047] At the step S15, the binarization unit 15 binarizes 
image data inputted from the character recognition unit 11 
and/or area separation unit 12 if the binarization process 
is set up as the post-process. More specifically, as an 
example, if the character images in the character areas are 
not be converted into the character code data , the binarization 
unit 15 binarizes the input image data according to the 
specified threshold value. The binarized character image 
data thus obtained will be added with the position information 
and color information and inputted into the synthesizing unit 
16. 

[0048] At the step S16, the synthesizing process will be 
conducted, in which the data from the character recognition 
unit 11, the bit map processing unit 13, the vector conversion 
unit 14, and the binarization unit 15 are synthesized and 
outputted. 

[0049] Next, the synthesizing output process at the step 
S16 will be described in detail using the flow chart of Fig. 
3. 

[0050] At the step S21 , it is judged whether the background 
input mode is set up or not. In case it is the background 
input mode (S21: Yes), the process of the step S3 1 is executed, 
while, in case it is not the background input mode (S21: No) , 
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the process of the step S22 is executed. 

[0051] At the step S31, the synthesizing unit 16 stores 
all the data inputted into the synthesizing unit 16, i.e., 
the background image data, into the memory 18 to complete 
the operation of the image processing unit 10. More 
specifically, for example, the background image data shown 
in Fig. 4 is stored in the memory 18. 

[0052] On the other hand, at the step S22 it is judged whether 
the image synthesizing-yes mode is set up. In case it is the 
image synthesizing-yes mode (822: Yes), the process of the 
step 823 is executed, while, in case it is the image 
|xf synthesizing-no mode (822: No), the process of the step 828 

O is executed. 

W [0053] At the step 823, the synthesizing unit 16 extracts 

the position information of circumscribing rectangles of all 
elements of the foreground image data and detects the colors 
of the background image data that correspond to those 
circumscribing rectangles, i.e., the background colors. 
More specifically, for example, it extracts the position 
information of a circumscribing rectangle for each individual 
character image, etc., in the foreground image data shown 
in Fig. 5, plugs said circumscribing rectangle 66 into the 
background image data shown in Fig. 6, and detects the 
background color within said circumscribing rectangle. The 
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background color is an average color of the colors of pixels 
in said circumscribing rectangle. The foreground image data 
are made up of character code data from the character 
recognition unit 11, vector data from the vector conversion 
unit 14, and binarized character image data from the 
binarization unit 15. 

[0054] At the step S24, the synthesizing unit 16 enters 
the colors of the foreground image data, i.e. , the foreground 
P colors, and the background colors, into the correlation table 

^ shown in Fig . 7 . This correlation table is stored in the memory 

y [0055] The foreground colors are expressed, for example, 

p by the RGB color system as (Rn, Gn, Bn) (n = 1, 2, ...) and 

m 

bj are stored into the foreground color column of the correlation 

.□ 

table. The background colors are expressed, for example, by 
the RGB color system as (RBnm, GBnm, BBnm) (n, m = 1, 2, «.) r 
which are correlated to the foreground color of the 
corresponding position and are stored into the background 
color column of the correlation table. 

[0056] Let the two foreground colors that are compared be 
expressed in the RGB color system as (Ri, Gi, Bi) and (Rj, 
G j , B j ) . If the K value for those colors obtained from the 
following formula (1) is lower than a specified threshold 
value; 
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K = (Ri - Rj) ^ + (Gi - Gj) ^ + (Bi - Bj) ^ (1) 
then, the two colors are judged to be approximately equal 
and the two foreground colors will be grouped together for 
storage in the memory. In other words , if another foreground 
color, for which the K value according to the formula (1) 
is smaller than the specified threshold value, exists in the 
correlation table, the background color is added to the 
existing foreground color group. Moreover, the first 
foreground color, which entered the correlation table, 
becomes the color of the foreground color group. However, 
it is also possible to select the average of the colors in 
a foreground color group as the color of the foreground color 
group. On the other hand, if no other foreground color which 
makes the K value less than the specified threshold value 
exists in the correlation table, a new foreground color group 
will be added to the table, and background colors will be 
inputted into the table relating to be correlated to it. 
[0057] At the step 525, the synthesizing unit 16 calculates 
the average value of all the background colors entered in 
correlation with each foreground color group . The calculated 
average background color is to be entered into the average 
background color column of the correlation table. 
[0058] The average background color is obtained for each 
coordinate as an average value of the background colors 



Q 



expressed by the coordinate value of the specified color system . 
More specifically, the average background color value can 
be calculated according to the following formula (2) as 
expressed, for example, in the RGB color system as (RAn, GAn, 
BAn) : 



Imax 

RAn= yRBnl/lmax 
1^1 



Imax 

GAn= yGBnl/lmax (2) 
1^1 



Imax 

BAn= ySBnl/lmax 
1^1 



y where Imax is the number of background colors correlated to 

□ the foreground color group. 

y [0059] At the step S26, the synthesizing unit 16 calculates 

O 

Q the color of the foreground image data after the adjustment, 

i.e., the adjusting color , based on the color of the foreground 
color group and the average background color calculated in 
correspondence to the color of the foreground color group. 
In other words, the synthesizing unit 16 calculates an 
adjusting color that makes the boundary between the foreground 
image data and the background image data recognizable for 
all colors of the background image data. 

[0060] The foreground colors and the average background 
colors of the correlation table are now color converted from 
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the RGB color system to the L*a*b* color system. The 
synthesizing unit 16 selects as the adjusting color a color 
that" maintains the hue of a foreground color and is located 
furthest from the corresponding average background color in 
distance on the color space of the L*a*b* color system. More 
specifically, when the foreground color, the average 
background color, and the adjusting color of the foreground 
image data are expressed in the L*a*b* color system as (Ln, 
an, bn), (LAn, aAn, bAn), and (Lnc, anc, bnc), the synthesizing 

^ unit 16 calculates the (Lnc, anc, bnc) that maximizes the 

y, value J in the following formulas : 

y J = (Lnc - LAn) ^ + (anc - aAn) ^ + (bnc - bAn) ^ 

□ H = bn/an 

yJ [0061] The method of calculating the adjusting color is 

Q 

Q not limited to the above method. Forexample, the synthesizing 

unit 16 can calculate the adjusting color by setting: 

anc = I an I when aAn < 0; anc = - [ an | when aAn ^ 0 ; 

bnc = I bn I when bAn < 0 ; bnc = - | bn | when bAn ^ 0 ; 
and maximizing Lnc. 

[0062] The adjusting colors (Lnc, anc, bnc) calculated in 
the L*a*b*color system are then color converted into the RGB 
color system, and the adjusting colors (Rnc, Gnc, Bnc) 
expressed in the RGB color system are then entered in the 
adjusting color column of the correlation table. 
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[0063] The colors (R, G, B) in the RGB color system are 
color converted into the colors (X, Y, Z) in the XYZ color 
system by means of the following formula (3), and further 
into the colors (L*^ a*, b* ) in the L*a*b* color system 
according to the formula (4). The color conversion in the 
opposite direction can be done using inverse function of the 
formulas (3) and (4). The 3x3 matrix of formula (3) shown 
below is a proper value of a device such as a printer or a 
scanner, and is prepared as a data for describing the 
characteristics such as the correspondence of the colors for 
color matching. 
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L*= 1 1 6 (Y/Yo) 1 6 

a*= 5 0 0 [ (X/Xo) (Y/Yo) '^'1 (4) 

b*= 2 0 0 [ (YXYo) (Z/Zo) '^'3 

[0064] At the step 321 , the synthesizing unit 16 replaces 
the color information (R, G, B) attached to the character 
code data, the vector data, and the binarized character image 
data as the foreground image data with the calculated adjusting 
color (Rnc, Gnc, Bnc) with reference to the correlation table. 
[0065] At the step S28 , the synthesizing unit 16 synthesizes 
the data from the character recognition unit 11, the bit map 



processing unit 13, the vector conversion unit 14, and the 
binarization unit 15. If the image synthesizing-yes mode is 
set up as the image synthesizing mode, the foreground image 
data and the background image data, i.e., its background, 
are synthesized. Both image data will be synthesized 
according to a predetermined internal format. 
[0066] At the step S29, the format conversion unit 17 
converts the image data synthesized according to the 
predetermined internal format into the output format 
specified by the user. 

[0067] At the step S30, the format conversion unit 17 
transmits the synthesized image data converted to the output 
format to the interface 3 4 connected to the image output unit 
50. 

[0068] Fig. 8 is a diagram showing an example of a 

synthesized image data produced by synthesizing the 
background image data shown in Fig. 4 and the foreground image 
data shown in Fig. 5. 

[0069] The background image data here consists of a green 
leaf section 61 and a brown background section 62 (see Fig. 
4) . The foreground image data consists of a black character 
section 63 , a red symbol section 64 , and a deep green character 
section 65 (see Fig. 5) . In the synthesized image data where 
both image data are synthesized, the black character section 



63 of the foreground image data is color-adjusted to a white 
character section 73, and the deep green character section 
65 is color-adjusted to red character section 75 • 
[0070] Therefore, each section of the foreground image data 
becomes recognizable against both the green leaf section 61 
and the brown background section 62 of the background image 
data, making them more legible- Moreover, although the 
character sections 63 and 65 of the foreground image data 
Q are both overlapping with both of different color sections 

^ 61 and 62 of the background image data, they are not adjusted 

^ i 

1^ to easily identifiable colors on the background image data, 

and the character sections , which appeared as approximately 
same colors on the foreground image data, are color-adjusted 

y into a uniform color- Thus, the designer's original 

□ 

□ color-coded information, e.g. , the information for the title, 

subtitle and keyword, will be maintained in the foreground 
image data. 

[0071] As can be seen from the above, the specific colors 
in the foreground image data having approximately same colors 
are adjusted to a uniform color, which makes the boundaries 
between the foreground image data and the background image 
data recognizable to all the colors of the background image 
data that serves as the background. Therefore, color-coding 
information such as the title expressed in a uniform color 
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on the foreground image data will not be lost from the 
synthesized image data obtained by synthesizing the 
foreground image data such as character images and its 
background, i.e., the background image data. Moreover, 
character images, etc., are easily legible against the 
background image data. 

[0072] The invention is not limited to the above-mentioned 
embodiment, but rather can be changed in various ways with 
the scope of the claims of the patent. 

[0073] For example, although it was described about a color 
copying system that functions as an image processing device 

y equipped with the image input unit 40 consisting of a color 

II 

□ scanner, etc. , and the image output unit 50 having the first 

UJ output unit 51 consisting of a color printer, etc., as well 

W as the second output unit 52 consisting of a computer having 

a monitor, etc., the invention is not limited to it. For 
example, the invention can be applied to a digital color copying 
machine that has an image input unit and an image output unit 
built into the machine. 

[0074] The invention also can be applied to a computer that 
functions as an image processing device. In this case, the 
image data can not only be prepared by the computer but also 
can be received from other information equipment such as a 
scanner or can be downloaded via a network. 
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[0075] Furthermore^ it often happens with presentation 
materials that different foreground image data are drawn over 
the same background image data so that multiple sheets can 
be prepared quickly . In such a case ^ it is possible to transmit 
the background image data to the printer only once while 
transmitting as many foreground images as needed for 
synthesizing both images at the printer's controller. 
[0076] Various means to constitute the image processing 



'Z device and the image processing method according to this 

uO 

^ invention can be realized by a dedicated hardware circuit 

m 

or a programmed computer. In realizing the invention by means 
LiJ of a programmed computer, the program for operating the 

computer can be provided by computer readable recording medium 
(e.g., flexible disk or CD-ROM). In this case, the program 
recorded on computer readable recording medium is normally 
transferred to the hard disk and stored. This program can 
be provided as independent application software or can be 
built into the software of the computer as a function of the 
computer. The program product in this invention includes the 
program itself as well as the computer readable recording 
medium on which the program is stored. 
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